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(54) Method and system for tracking screen activities in online transactions 



(57) A digital evidence system is disclosed. Accord- 
ing to one embodiment, the digital evidence system 
records pertinent screen activities such as data entry in- 
to an electronic form, verification of a data transfer and 
a purchase order by capturing images of displayed web 



pages requiring interactions from the user. The captured 
images are transmitted back to the business and can be 
retrieved according to a query about any interaction so 
that the online business could determine what has hap- 
pened on the user side when a transaction was engaged 
with the online business. 
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Description 

[0001] The present invention is related to the area of 
electronic commerce over the Internet and more partic- 
ularly related to an electronic method and system for 
tracking screen activities conducted by a user when the 
user enters transactions with an online business over 
the Internet. The screen activities are recorded in imag- 
es as digital evidence that may be retrieved for verifica- 
tion of the transactions when a need arises. 
[0002] The Internet is a data communication network 
of interconnected computers and computer networks 
around the world and is rapidly evolving to the point 
where it combines elements of telecommunications, 
computing, broadcasting, publishing, commerce, and 
information services into a revolutionary new business 
infrastructure. The economy on the Internet is growing 
in every aspect of life, a wide range of businesses in- 
cluding stock trading, commodities, products, and serv- 
ices ordering are all via the Internet. However, the mush- 
rooming Internet-based electronic commerce (e-com- 
merce) offers opportunities for cyber crimes including 
credit card frauds, virus attacks, and irrational disputes, 
which could cost online businesses millions of dollars or 
more each year. 

[0003] Many online businesses, such as www.ama- 
zon.com , require a buyer to submit electronically per- 
sonalized information before any transactions take 
place. The personalized information including the buy- 
er's address and payment information (i.e. a credit card 
number) is provided to a merchant as to where to ship 
and how to charge for purchased items. The merchant 
however has no actual means to verify that the person- 
alized information submitted is authenticated. For ex- 
ample, the credit card number could be a stolen one or 
the entire transaction is fraudulent. In other cases, a 
buyer may dispute that a purchased item that has been 
delivered from the merchant is not what the buyer orig- 
inally selected in the transaction. Without any evidence 
how the buyer entered the order, the merchant often has 
to admit that a mistake was made on the merchant side 
by taking the item back and bearing all associated costs. 
As a result, the online transaction overhead for the mer- 
chant goes higher and the Internet economy could be 
potentially hampered. 

[0004] There is, therefore, a great need for a generic 
solution that provides a merchant with an effective 
means to study how a transaction is engaged on the 
buyer's side so that necessary steps could be taken to 
fairly resolve any disputes or overcome any fraudulent 
transactions to minimize the online transaction over- 
head. 

[0005] The present invention has been made in con- 
sideration of the above described problems and needs 
and has particular applications to electronic commerce 
over the Internet. One of the key features of the present 
invention is to provide a digital evidence system that can 
support an online business to make a justified decision 



or take necessary measures whenever there is a need. 
The digital evidence system records pertinent screen 
activities such as data entry into an electronic form, ver- 
ification of a data transfer and a purchase order by cap- 

5 turing images of displayed web pages requiring interac- 
tions from the user. The captured images are transmit- 
ted back to the business and can be retrieved according 
to a query about any interaction by the user so that the 
online business could determine what has happened on 

10 the user side when a transaction was engaged with the 
online business. 

[0006] In a first aspect the present invention therefore 
provides a method for tracking screen activities en- 
gaged by a user with a client device for online transac- 
ts tions over a data network, the method comprising: re- 
quiring an entry from the user to a display after the user 
decides to enter a transaction over the data network with 
a provider; receiving image data from the data network, 
the image data recording the display including the entry 

20 by the user; and storing the image data associated with 
the transaction in a memory space. 
[0007] In a second aspect the present invention pro- 
vides a method for tracking screen activities engaged 
by a user with a client device for online transactions over 

25 a data network, the method comprising: requiring inter- 
actions from the user to one or more displays after the 
user decides to enter a transaction over the data net- 
work with a provider; determining if the transaction is 
complete; receiving image data from the data network 

30 if the transaction is complete, the image data recording 
the one or more displays including the interactions by 
the user; and storing the image data associated with the 
transaction in a memory space. 
[0008] In a third aspect the present invention provides 

35 a method for tracking screen activities engaged by a us- 
er with a client device for online transactions over a data 
network, the method comprising: causing the client de- 
vice to display an electronic document thereon, the elec- 
tronic document demanding an interaction from the user 

40 before the user is allowed to proceed with a transaction; 
and receiving image data over the data network from 
the client device; the image data representing an image 
of the electronic document including the interaction per- 
formed by the user, wherein the image is captured lo- 

45 cally in the client device as soon as the user permits to 
release the electronic document including the interac- 
tion. 

[0009] In a fourth aspect the present invention pro- 
vides a method for tracking screen activities engaged 

so by a user with a cl ient device for online transactions over 
a data network, the method comprising: executing a 
browsing application to contact a server device over the 
data network by sending out a request; displaying a 
page from the server device after the server device re- 

55 sponds to the request; the page requiring inputs from 
the user; activating a screen capturing application to 
capture an image of the page including the inputs as 
soon as the user authorizes to release the inputs to the 
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server device; and transmitting image data representing 
the image together with the inputs over the data network 
to the server device. 

[0010] In a fifth aspect the present invention provides 
a method for tracking screen activities engaged by a us- 
er with a client device for online transactions over a data 
network, the method comprising: receiving respectively 
a sequence of web pages from a server device after the 
server device is contacted for conducting a transaction 
over the data network; displaying individually the web 
pages for inputs from the user; accepting the inputs by 
showing where the inputs are entered in the web pages; 
activating a screen capturing application to capture an 
image of each of the web pages including the inputs eve- 
ry time the user authorizes to release the inputs to one 
of the web pages to the server device; storing images 
of the web pages including the inputs in a local memory 
space; and releasing the images to the server device 
after the transaction is complete. 
[0011] In a sixth aspect the present invention provides 
a method for tracking screen activities engaged by a us- 
er with a client device for online transactions over a data 
network, the method comprising: receiving a web page 
from a server device after the server device is contacted 
for conducting a transaction over the data network; dis- 
playing the web page for an input from the user; accept- 
ing the input by showing where the input is entered in 
the web page; activating a screen capturing application 
to capture an image of the web page including the input 
as soon as the user authorizes to release the input to 
the server device; and transmitting image data repre- 
senting the image together with the input over the data 
network to the server device. 

[0012] In a seventh aspect the present invention pro- 
vides a system for tracking screen activities engaged by 
a user with a client device for online transactions over 
a data network, the system comprising: a server device 
operated by a provider; the server including a digital ev- 
idence module that receives image data from the client 
device; the client device executing a browsing applica- 
tion that permits the client device to conduct a transac- 
tion over the data network with the server device; where- 
in the image data represents an image of a display in- 
cluding a web page from the server device, the web 
page requiring an input from the user; the client device 
executing a screen capturing application to generate the 
image and convert the image to the image data after the 
user completes the input; wherein the browsing appli- 
cation causes the image data to be transmitted to the 
server device; and wherein the digital evidence module 
associates the image data with the transaction. 
[0013] In a eighth aspect the present invention pro- 
vides a system for tracking screen activities engaged by 
a user with a client device for online transactions over 
a data network, the system comprising: a processor; a 
first memory space for a browsing application, when ex- 
ecuted by the processor, to permit the user to conduct 
the online transactions over the data network with a 



server; a second memory space for code of a screen 
capturing module, the screen capturing module activat- 
ed only and executed by the processor when the user 
decides to release to the server one or more inputs en- 

5 tered in a web page by the user; the screen capturing 
module capturing an image of the web page including 
the one or more inputs; and wherein the browsing ap- 
plication causes the image together with the one or more 
inputs to be transmitted to the server. 

w [0014] In a ninth aspect the present invention pro- 
vides a system for tracking screen activities engaged by 
a user with a client device for online transactions over 
a data network, the system comprising: a processor; a 
first memory space for a browsing application, when ex- 

15 ecuted by the processor, to permit the user to conduct 
the online transactions over the data network with a 
server; a second memory space for caching images of 
screen displays, the images captured successively by 
a screen capturing application when the user decides to 

20 release to the server one or more inputs entered in each 
of the screen displays; wherein the captured images in- 
clude the one or more inputs; and wherein the browsing 
application causes the cached images to be transmitted 
to the server. 

25 [0015] In a tenth aspect the present invention pro- 
vides a system for tracking screen activities engaged by 
a user with a client device for online transactions over 
a data network, the system comprising: a processor; a 
first memory space for an online transaction module, 

30 when executed by the processor, to permit the online 
transactions with the client device over the data net- 
work, the online transaction module causing the client 
device to display a page requiring one or more inputs 
from the user; a second memory space for a digital ev- 

35 idence module, when executed by the processor, to 
work synchronously with the online transaction module; 
and wherein the digital evidence module receives image 
data representing an image of the page including the 
one or more inputs from the client device. 

40 [0016] In an eleventh aspect the present invention 
provides a computer program for tracking screen activ- 
ities engaged by a user with a client device for online 
transactions over a data network, the computer program 
comprising: program code for requiring an entry from the 

^5 user to a display after the user decides to enter a trans- 
action over the data network with a provider; program 
code for receiving image data from the data network, 
the image data recording the display including the entry 
by the user; and program code for storing the image data 

50 associated with the transaction in a memory space. 
[0017] In a twelfth aspect the present invention pro- 
vides a computer program for tracking screen activities 
engaged by a user with a client device for online trans- 
actions over a data network, the computer program 

55 comprising: program code for requiring interactions 
from the user to one or more displays after the user de- 
cides to enter a transaction over the data network with 
a provider; program code for determining if the transac- 
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tion is complete; program code for receiving image data 
from the data network if the transaction is complete, the 
image data recording the one or more displays including 
the interactions by the user; and program code for stor- 
ing the image data associated with the transaction in a 
memory space. 

[0018] In a thirteenth aspect the present invention 
provides a computer program for tracking screen activ- 
ities engaged by a user with a client device for online 
transactions over a data network, the computer program 
comprising: program code for causing the client device 
to display an electronic document thereon, the electron- 
ic document demanding an interaction from the user be- 
fore the user is allowed to proceed with a transaction; 
and program code for receiving image data over the da- 
ta network from the client device; the image data repre- 
senting an image of the electronic document including 
the interaction performed by the user, wherein the image 
is captured locally in the client device as soon as the 
user permits to release the electronic document includ- 
ing the interaction. 

[0019] In a fourteenth aspect the present invention 
provides a computer program for tracking screen activ- 
ities engaged by a user with a client device for online 
transactions over a data network, the computer program 
comprising: program code for executing a browsing ap- 
plication to contact a server device over the data net- 
work by sending out a request; program code for dis- 
playing a page from the server device after the server 
device responds to the request; the page requiring in- 
puts from the user; program code for activating a screen 
capturing application to capture an image of the page 
including the inputs as soon as the user authorizes to 
release the inputs to the server device; and program 
code for transmitting image data representing the image 
together with the inputs over the data network to the 
server device. 

[0020] In a fifteenth aspect the present invention pro- 
vides a computer program for tracking screen activities 
engaged by a user with a client device for online trans- 
actions over a data network, the computer program 
comprising: program code for receiving respectively a 
sequence of web pages from a server device after the 
server device is contacted for conducting a transaction 
over the data network; program code for displaying in- 
dividually the web pages for inputs from the user; pro- 
gram code for accepting the inputs by showing where 
the inputs are entered in the web pages; program code 
for activating a screen capturing application to capture 
an image of each of the web pages including the inputs 
every time the user authorizes to release the inputs to 
one of the web pages to the server device; program 
code for storing images of the web pages including the 
inputs in a local memory space; and program code for 
releasing the images to the server device afterthe trans- 
action is complete. 

[0021] In a sixteenth aspect the present invention pro- 
vides a computer program for tracking screen activities 



engaged by a user with a client device for online trans- 
actions over a data network, the computer program 
comprising: program code for receiving a web page from 
a server device after the server device is contacted for 

5 conducting a transaction over the data network; pro- 
gram code for displaying the web page for an input from 
the user; program code for accepting the input by show- 
ing where the input is entered in the web page; program 
code for activating a screen capturing application to cap- 

10 ture an image of the web page including the input as 
soon as the user authorizes to release the input to the 
server device; and program code for transmitting image 
data representing the image together with the input over 
the data network to the server device. 

15 [0022] Ideally, a screen capturing module is activated 
to capture an image or images of displays whenever a 
user completes an interaction. The interaction may in- 
clude data entry, verification, and viewing proof of a doc- 
ument. Specifically, an image of the display including the 

20 interaction by the user is captured and subsequently 
transmitted to the server. On the server side, a screen 
management system employing a digital evidence mod- 
ule configured to cause images of the displays to be cap- 
tured manages the images and associates the images 

25 with a respective account designated to the user or a 
transaction identification so that it makes possible to lat- 
er examine the images when there is a query about the 
user or the transaction. 

[0023] The present invention can be implemented in 

30 numerous forms. Different implementations of the 
present invention yield one or more of the following ad- 
vantages. One advantage of the invention is that an ef- 
fective digital system is provided to online businesses 
to track screen activities conducted by users who en- 

35 gage in various transactions with the businesses so that 
the businesses can take necessary steps or make jus- 
tified decisions when disputes or frauds arise. Another 
advantage of the invention is to activate the screen cap- 
turing process automatically when a user authorizes to 

to release the user-provided data. The screen capturing 
process can be performed explicitly or implicitly depend- 
ing on a particular implementation. Still another advan- 
tage of the present invention is the management of the 
screen images that need to be retrieved only when there 

45 is a need. 

[0024] Accordingly one of the objects of the present 
inventions is to provide a digital evidence system that 
can be used effectively to support the online business 
of many kinds. 

so [0025] Objects and advantage together with the fore- 
going are attained in the exercise of the invention in the 
following description and resulting in the embodiments 
illustrated in the accompanying drawings. 
[0026] These and other features, aspects, and advan- 

55 tages of the present invention will become better under- 
stood with regard to the following description, appended 
claims, and accompanying drawings where: 
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Figure 1 shows a system configuration in which the 
present invention may be practiced; 
Figure 2A shows a functional diagram of a provid- 
er's server according to one embodiment of the 
present invention; 

Figure 2B shows a process flowchart of a digital 
evidence module used in the provider's server of 
Figure 2A; 

Figure 2C shows a functional block diagram of a 
digital evidence module according to one embodi- 
ment used in an exemplary system; 
Figure 3A illustrates an exemplary web page re- 
quiring a user to enter quantity information to be re- 
leased to a server; 

Figure 3B illustrates another exemplary web page 
requiring a user to enter personal information to be 
released to a server; 

Figure 4A shows a process flowchart of the present 
invention in the client device according to one em- 
bodiment; and 

Figure 4B shows a process flowchart of the present 
invention in the client device according to another 
embodiment. 

[0027] According to one aspect of the present inven- 
tion, a method and system for tracking screen activities 
performed by a user on a client device communicating 
over a data network with a server device are disclosed. 
Such client device may include, but not be limited to, a 
personal computer, a palm-sized device and a wireless 
mobile device, all including a display screen and is ca- 
pable of two-way communication. The data network can 
be the Internet and the server device may be any com- 
puting device operated by a provider that may include, 
but not be limited to, a retailer, a merchant, an online 
business, a service provider or a brokerage firm. With 
the employment of the present invention, the provider 
can be supported with a digital evidence system that 
records anything entered by the user and hence make 
justified decisions with respect to any possible disputes 
or frauds. 

[0028] in the following detailed description of the 
present invention, numerous specific details are set 
forth in order to provide a thorough understanding of the 
present invention. However, it will become obvious to 
those skilled in the art that the present invention may be 
practiced without these specific details. In other instanc- 
es, well known methods, procedures, components, and 
circuitry have not been described in detail to avoid un- 
necessarily obscuring aspects of the present invention. 
The detailed description is presented largely in terms of 
procedures, logic blocks, processing, and other symbol- 
ic representations that directly or indirectly resemble the 
operations of data processing devices coupled to net- 
works. These process descriptions and representations 
are the means used by those experienced or skilled in 
the art to most effectively convey the substance of their 
work to others skilled in the art. The method along with 



the system to be described in detail below is a self-con- 
sistent sequence of processes leading to one or more 
desired results. It proves convenient at times, principally 
for reasons of common usage, to refer to data as mes- 
s sages, requests, values, elements, symbols, terms, 
numbers, or the like. It should be borne in mind that all 
of these similar terms are to be associated with the ap- 
propriate physical quantities and are merely convenient 
labels applied to these quantities. 
w [0029] Referring now to the drawings, in which like nu- 
merals refer to like parts throughout the several views. 
Figure 1 shows a system configuration 100 in which the 
present invention may be practiced. As used herein, a 
service, a product provider, or simply a provider repre- 
ss sents one of numerous businesses, entities, merchants 
or retailers that offer various products, applications and 
services online. Examples of such a provider may in- 
clude www.amazon.com that offers items from books to 
consumer electronics, www.ameritrade.com that offers 
20 services to allow investors to trade stocks online, and 
www.evc-aips.com that offers home study services for 
traffic education. 

[0030] As shown in the figure a provider 102, through 
a server computer, is coupled to a 'data network 104 that 

25 may be the Internet or a network of private and/or public 
networks. To facilitate online transactions between the 
provider and a user (e.g. a consumer), a third party entity 
106, also through a server computer, is coupled to net- 
work 104. Third party entity 106 may represent a finan- 

30 cial or authorized institution, such as a bank, a credit 
card authority or an agent, that provides a mechanism 
for the provider to verify various information submitted 
by the buyer, to collect charges for items purchased by 
the user or to electronically proof anything required by 

35 the provider for the user. 

[0031] As used herein, a transaction means an elec- 
tronic agreement mutually agreed between two entities. 
For example, a transaction occurs when a provider ac- 
cepts an order of certain items from a buyer. Likewise a 

40 transaction is deemed to occur when pertinent data/in- 
formation entered/provided by one side is released to 
the other side. In general a transaction means a se- 
quence of information exchange and related work (such 
as database updating) that is treated as a unit for the 

45 purposes of satisfying a request and for ensuring data- 
base integrity. For a transaction to be completed and 
database changes to occur, a transaction has to be com- 
pleted in its entirety. One of the common transactions 
over the Internet is a catalog merchandise order by a 

50 customer. When the order is received on the server side, 
the order transaction involves checking an inventory da- 
tabase, confirming that the item is available, placing the 
order, and confirming that the order has been placed 
and the expected time of shipment. If the order is 

55 deemed as a single transaction, then all of the steps 
must be completed and validated before the transaction 
is successful and the database is actually changed to 
reflect the new order. 
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[0032] To follow up with the above explanation, serv- 
ices provided online by an entity may include, but not 
be limited to, stock trading, various education programs, 
various registrations, electronic tax filing, voting, online 
gambling and testing, and typically involve data entry or 
pertinent information to be supplied by a user. For ex- 
ample, in stock trading, a user must enter a stock sym- 
bol, a number of shares and sometimes a share price 
when a trading web page is displayed. Further, it is de- 
fined without the loss of generality that an interaction 
involves activities performed by a user with respect to a 
displayed page. In the above example, the stock infor- 
mation entered to the trading web page by the user is 
deemed as an interaction when the trading web page is 
completed. One of the key features in the present inven- 
tion is to record the interaction as a digital evidence so 
that the interaction can be retrieved later on when there 
is a need to review it. 

[0033] For illustration purpose, a user 108 communi- 
cates with provider 102 through a computing device, re- 
ferred to as a client device, that may include, but not be 
limited to, a personal computer, a palm-size computing 
device, a wireless mobile device and a personal data 
assistant (PDA), all having a display screen and capable 
of two-way data communications with data network 1 04 
for ordering electronically products or services from an 
online business, receiving a confirmation, or making 
payment thereafter. 

[0034] To facilitate the description of the present in- 
vention, it is defined herein a display screen or a screen 
is the physical display apparatus in a device, such as a 
1 5 inch CRT or LCD monitor commonly seen with a per- 
sonal computer. A screen display, a displayed web 
page, a displayed window or simply a display is an im- 
age presented on the display screen. For example, a file 
that constitutes a display may be an HTML file, wherein 
HTML stands for HyperText Markup Language, an im- 
age thereof appears on a display screen when the file 
is read or executed by a browser. 
[0035] For completeness, a service provider 110, also 
through a respective server computer, is coupled be- 
tween network 1 08 and sub-network 112. Typically, sub- 
network 112 that may be part of network 108 or a local 
area network (LAN) belonging to a community serves 
its own community comprising a number of set-top box- 
es 114, each representing an access to the service pro- 
vider from a user or a consumer. In general, each of set- 
top boxes 114 may be a personal computer, a palm-size 
computing device, a mobile device or a cable control box 
facilitating an Internet server via a cable. Via service pro- 
vider 110 that may include an Internet service provider, 
a wireless carrier, a cable service provider, or any data 
service provider, each of set-top boxes 114 can com- 
municate with provider 102 for ordering electronically a 
product or a certain type of services from provider 102. 
[0036] Referring now to Figure 2A, there is shown a 
functional diagram of a provider's server 200 according 
to one embodiment of the present invention. Server de- 



vice 200 may be a personal computer or a workstation 
used by provider 102 of Figure 1 to offer products for 
sales or services online and typically loaded with a serv- 
er module 202 that may be configured to include an on- 

5 line transaction module 204 and a digital evidence mod- 
ule 206. Online transaction module 204 may be any one 
of the commercially available or customized applica- 
tions that facilitate transactions with a client device over 
a network. One exemplary online transaction module 

10 204 may be the one used at www.officemax.com that 
permits a user to select and make an order of an item 
for sale, 

[0037] Figures 3A-3B show respectively one of the 
exemplary functions that online transaction module 204 

15 performs. As shown in Figure 3A, after a sequence of 
selection, a buyer finally determines to purchase a com- 
mercial file cabinet. A web page 300 is displayed on a 
screen of a computing device for the buyer and requires 
the buyer to enter a quantity to purchase. The computing 

20 device may be executing a browser such as a Netscape 
Communicator from www.netscape.com or an Internet 
browser from www.microsoft.com. To proceed with the 
purchase, web page 300 displayed by the browser de- 
mands that the buyer has to click "Checkout" 304 to pro- 

25 ceed with the purchase. As soon as "Checkout" 304 is 
clicked, the browser generates a message including the 
quantity information and sends the message to online 
transaction module 204 that subsequently stores the 
quantity information, preferably, associated with atrans- 

30 action identification or a user account designated to the 
buyer. If online transaction module 204 determines that 
additional information about the buyer is needed, a re- 
sponse to the message is then sent out from the server 
device and causes the browser to display a next web 

35 page 310 as shown in Figure 3B. Web page 310 as dis- 
played requires the buyer to provide the required addi- 
tional information so that online transaction module 204 
is ensured to fulfill the purchase order. 
[0038] In the above example, online transaction mod- v 

40 ule 204 captures typically text information entered by the 
buyer but could be deceived by a purchase dispute. For 
example, wrong quantity information was entered at 302 
of Figure 3A, and the purchase dispute could acclaim 
that online transaction module 204 has misinterpreted 

45 the message and subsequently causes the provider to 
be responsible for all costs associated with the wrong 
quantity information. The above example illustrates only 
one aspect of how online transaction module 204 could 
be subject to a possible fraud. There are many similar 

so or more damaging incidents, collectively referred to as 
the cyber crime or fraud over the Internet. One of the 
key advantages and benefits in the present invention is 
to minimize such disputes or frauds by providing digital 
evidence related to the user's screen activities. 

55 [0039] Referring back to Figure 2A, digital evidence 
module 206 controls screen capturing process. Figure 
2B shows a process flowchart of digital evidence mod- 
ule 206 according to one embodiment of the present in- 
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vention. When a client or computing device used by a 
buyer is caused to establish a data link with a provider's 
server and proceeds with a purchase order, online trans- 
action module 204 retrieves an account for the buyer or 
establishes an account for the buyer if the buyer is new 
to the provider. The account may be updated with infor- 
mation received from the data link as to where and how 
the computing device has requested such link to be es- 
tablished thereto. The information may include an ad- 
dress identifier of the computing device. One exemplary 
address identifier is an Internet protocol (IP) address 
from which the computing device communicates with 
the server. The IP address may be assigned by an iSP 
when the computing device is logged onto the ISP for 
communication over the Internet. 
[0040] The information may further include what 
browsing tool or application (e.g. a browser) the com- 
puting device is using to interact with the server. One of 
the browsing tools may be an Internet browser that gen- 
erates requests to contact a server with its own identity 
therein. With the data link established between a client 
device and the server device and after the user browsed 
to an appropriate site or page (e.g. to engage in a trans- 
action), at 220 a web page requiring user's interaction 
is sent to the computing device. The interaction may in- 
clude a data entry, a click on a predefined area or met- 
aphor or a click to cause a display of an electronic doc- 
ument as an acknowledgement. 
[0041] At 222, it is to determine if the page just sent 
to the client has any embedded screen capture or cap- 
turing application (capability). The screen capturing ap- 
plication is preferably a computer program or an appli- 
cation, when executed, capable of recording one or 
more selected window screen displays or the entire dis- 
play screen of the client device. One exemplary screen 
capturing product, called "FullShot", has similar con- 
cepts in the screen capturing application and is offered 
from Inbit, Inc. located at 1340 South De Anza Boule- 
vard, Suite 202, in San Jose of California 95129. Full- 
Shot permits a user to selectively capture an entire dis- 
play or one or more displayed windows in the display 
screen. 

[0042] If it is determined that the client device does 
not have any screen capturing application, a provider's 
preferred screen capturing application can be explicitly 
uploaded into the client device (i.e. the client device 
downloads the provider's preferred screen capturing ap- 
plication). The preferred screen capturing application 
may be a thin and modified version of FullShot and per- 
forms only screen capturing function. For example, web 
page 300 of Figure 3A is a displayed window that can 
be captured by executing the preferred screen capturing 
application. Given the description of what the preferred 
screen capturing application intends to do, those skilled 
in the art will have no difficult to implement such screen 
capturing application. 

[0043] In one preferred implementation, the screen 
capturing application is implemented as an applet that 



is embedded in a web page. When the web page is load- 
ed in a client device and caused to be displayed thereon, 
the applet is activated to capture the web page with us- 
er's inputs as soon as the user clicks on a designated 
5 area or link (hence leading to "Y" of 222 of Figure 2B). 
[0044] As described above, a message is generated 
when the buyer clicks "CHECKOUT* 304 of Figure 3A, 
the message includes what the buyer entered. One of 
the features in the present invention is to capture the 
10 displayed window (e.g. displayed web page 300) when 
the buyer clicks "CHECKOUT' 304. The preferred 
screen capturing application is executed when the buyer 
clicks "CHECKOUT' 304. In other words, an execution 
command is placed inherently with a clickable area, 
15 such as a hyperlink, an "hot" area or metaphor, a soft 
key or a designated sign. The captured displayed win- 
dow is generally in a digital image format that can be 
cached in a local memory or sent out with the message. 
At 226, such images are successively received from the 
client device every time the buyer is asked to click a pre- 
defined area in the web page such as "submit", or 
"checkout", etc., or collectively received as a set of im- 
age data from the client device when the buyer finishes 
the transaction. 

[0045] In some applications, each of captured images 
related to a transaction is preferred to be sent out indi- 
vidually, for example, together with data provided to a 
particular page. In other applications, each of captured 
images is initially cached in a local memory and the 
stored images are collectively sent out as soon as a 
transaction is complete. 

[0046] Generally, the images received from a client 
device are kept as a digital evidence of the screen ac- 
tivities in a database or a storage device and preferably 
associated respectively with transaction identifications 
or user accounts for future references as illustrated in 
Figure 2C in which digital evidence module 250 inter- 
faces with a storage device or a database 252. In a case 
that a purchase dispute arises from a buyer, a set of 
screen images can be retrieved at 236 from the data- 
base or the storage device according to a query 254 re- 
lated to a user account or a transaction ID involved in 
the purchase dispute. The set of images can be sequen- 
tially displayed at 238 in an order that the buyer pro- 
ceeded, the provider can determine how or what screen 
activities engaged by the buyer at the time the related 
transactions occurred and makes a justified decision by 
examining the digital evidence at 240. 
[0047] Referring to Figures 4A-B, there are respec- 
tively shown a process flowchart of the client device ac- 
cording to one embodiment of the present invention. In 
accordance with the description above, Figures 4A-B 
shall be understood in conjunction with preceding fig- 
ures. At 402 of Figure 4 A, the client device establishes 
a data link with the provider's server. The data link pref- 
erably includes a secure session when personal or fi- 
nancial information is transmitted over an open data net- 
work, such as the Internet. The two of the popular brows- 
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ers, Netscape Communicator and Internet Explorer, all 
provide such mechanism to establish a secure session 
with the server when the server requires private data by 
sending a flag signal to activate the secure session. Typ- 
ically a user is offered to select any items or services of 
interest or browse the entire site of the provider. When 
the buyer decides to enter a transaction for one or more 
items or services, the browser prompts to the buyer a 
screen for personalized information that may include 
number of shares of a particular stock to trade at a pre- 
defined price, entry of an electronic form and decisions 
on any electronic agreement (often seen from a broker- 
age web site), a particular product and a type of service. 
One example of such screens may be web page 310 of 
Figure 3B in which a lot of personalized data must be 
entered. As soon as the user permits a submission of 
the data by clicking a "continue", the screen is instantly 
captured at 408 and the image thereof is associated with 
the data just entered by the user. At 410, the captured 
images are processed, One of the reasons for process- 
ing the images is to make the transmission of the images 
to the server more efficiently so that the user does not 
have to wait for too long before seeing a next screen. 
As it is well known to those skilled in the art that the 
screen images, especially those from HTML-based web 
pages, comprise large homogenous areas that are ap- 
propriate for some of the image compression tech- 
niques such as Run-Length-Encoding or LZW compres- 
sion. The resultant compressed image may have a 
much smaller size than the original does, thus making 
the transmission of the captured screen images to the 
server at 41 2 fast enough so that the user would not feel 
prolonged latency resulting from the transmission of the 
screen images. 

[0048] Referring now to Figure 4B, there is shown a 
process flowchart modified from Figure 4A. After 
screen capturing application is activated at 408 of Fig- 
ure 4A, the captured images are cached in a local mem- 
ory at 420. The captured images may be a processed 
version of the original images depending on an imple- 
mentation preference. At 420, it needs to determine if 
the current transaction is completed. This may be 
achieved by a message received from a server. If the 
transaction is incomplete, the user is required to contin- 
ue with the transaction. When it is determined that the 
current transaction is complete at 422, all the captured 
images in the local memory are processed for transmis- 
sion at 424. 

[0049] One of the differences between processes in 
Figure 4A and Figure 4B is how the captured image is 
handled depending on application needs. In many pur- 
chase order transactions, Figure 4B may be preferable 
as the captured images are not transmitted to the server 
till a buyer completes an order. In some applications, 
such as a proof of viewing a document, the captured 
image shall be released immediately to a server that 
seeks the proof. 

[0050] The flowchart processes respectively in Fig- 



ures 2A and 4A-4B along with the detailed description 
have made it evident to those skilled in art that the 
present invention can be implemented in numerous 
forms such as a system, a method or a computer read- 

5 able medium. The present invention is preferably inte- 
grated with a browser or a browsing application so that 
no additional procedures are required from users. In one 
implementation, the present invention is implemented 
as a plug-in that can be downloaded into a client device. 

10 in another implementation, the present invention is im- 
plemented as an applet or an executable application 
that goes with some of the web pages requiring user's 
inputs. Alternatively, an implementation of the present 
invention may be downloaded from a predesignated 

15 server when a client device communicates with a pro- 
vider's server and is caused to do so. 
[0051] Different implementations of the present in- 
vention yield one or more of the following advantages. 
One advantage of the invention is that an effective digital 

20 system is provided to online businesses to track screen 
activities conducted by users who engage in various 
transactions with the businesses so that the businesses 
can take necessary steps and make a justified decision 
when disputes or frauds arise. Another advantage of the 

25 invention is to activate the screen capturing process au- 
tomatically when a user authorizes to release the user- 
provided data. The screen capturing process can be 
performed explicitly or implicitly depending on a partic- 
ular implementation. Still another advantage of the 

30 present invention is the management of the screen im- 
ages that need to be retrieved only when there is a need. 
[0052] The present invention has been described in 
sufficient details with a certain degree of particularity. It 
is understood to those skilled in the art that the present 

35 disclosure of embodiments has been made by way of 
examples only and that numerous changes in the ar- 
rangement and combination of parts may be resorted 
without departing from the spirit and scope of the inven- 
tion as claimed. For example, the present invention may 

40 be practiced independently from an online transaction 
module and can be used to capture displayed pages for 
collecting evidences or updating a digital evidence sys- 
tem. Accordingly, the scope of the present invention is 
defined by the appended claims rather than the forego- 
es jng description of embodiments. 



Claims 

50 1. A method for tracking screen activities engaged by 
a user with a client device for online transactions 
over a data network, the method comprising: 

requiring an entry from the user to a display af- 
55 ter the user decides to enter a transaction over 

the data network with a provider; 
receiving image data from the data network, the 
image data recording the display including the 
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entry by the user; and 

storing the image data associated with the 
transaction in a memory space. 

2. The method of claim 1, wherein the step of requiring 
an entry comprises: 

causing the client device to display a form that 

must be filled by the user; and 

causing the client device to activate a screen 

capturing application to take an image of the 

form. 

3. A method for tracking screen activities engaged by 
a user with a client device for online transactions 
over a data network, the method comprising: 

requiring interactions from the user to one or 
more displays after the user decides to enter a 
transaction over the data network with a provid- 
er; 

determining if the transaction is complete; 
receiving image data from the data network if 
the transaction is complete, the image data re- 
cording the one or more displays including the 
interactions by the user; and 
storing the image data associated with the 
transaction in a memory space. 

4. The method of claim 3; wherein the step of requiring 
interactions from, the user comprises: 

causing the client device to capture an image 
of each of the one or more displays including 
the interactions by the user; and 
causing the client device to store the image in 
a local memory thereof. 

5. The method as claimed in any of the preceding 
claims, wherein the display is one of (i) an electronic 
form requiring one or more selections from the user, 
(ii) a service agreement requiring at least one elec- 
tronic signature from the user, (iii) a purchase order 
requiring at least quantity information, and (iv) a dis- 
played page requiring inputs from the user. 

6. The method as claimed in any of the preceding 
claims; wherein the image data is a compressed 
version of the image so that the transmission of the 
image over the data network can be efficient. 

7. A method for tracking screen activities engaged by 
a user with a client device for online transactions 
over a data network, the method comprising: 

causing the client device to display an electron- 
ic document thereon, the electronic document 
demanding an interaction from the user before 



the user is allowed to proceed with a transac- 
tion; and 

receiving image data over the data network 
from the client device; the image data repre- 
5 senting an image of the electronic document in- 

cluding the interaction performed by the user, 
wherein the image is captured locally in the cli- 
ent device as soon as the user permits to re- 
lease the electronic document including the in- 
fo teraction. 

8. The method of claim 7 further comprising associat- 
ing the image data with an account designated to 
the user so that the image data can be retrieved 

15 whenever there is a query about the user or the 
transaction. 

9. The method of claims 7 or 8, further comprising as- 
sociating the image data with the transaction so that 

20 the image data can be retrieved whenever there is 
a query about the transaction. 

10. The method as claimed in any of claims 7 to 9, 
wherein the interaction includes one or more of (i) 

25 data entry by the user to the electronic document, 
(ii) a click on a designated area of the electronic 
document and (iii) a click to display a new electronic 
document. 
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(54) Method and system lor tracking screen activities in online transactions 



(57) A digital evidence system is disclosed. Accord- 
ing to one embodiment, the digital evidence system 
records pertinent screen activities such as data entry in- 
to an electronic form, verification of a data transfer and 
a purchase order by capturing images of displayed web 



pages requiring interactions from the user. The captured 
images are transmitted back to the business and can be 
retrieved according to a query about any interaction so 
that the online business could determine what has hap- 
pened on the userside when a transaction was engaged 
with the online business. 
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